Boot Troubleshooting Labs

1. Selecting a Boot Target

Lab Resources

Machines: server1

Goal

  • A system booted in to different targets

In this lab, you will configure your server1 system to boot in to different targets.

  1. From the desktop1 system reset your server1 system.

    1. Configure your server1 system to boot in to the multi-user target by default.

      Do not reboot your server1 system right now, as it will make your graphical console unavailable.

      [root@server1 ~]# systemctl set-default multi-user.target
rm '/etc/systemd/system/default.target'
ln -s '/usr/lib/systemd/system/multi-user.target' '/etc/systemd/system/default.target'

    2. Verify that multi-user is now indeed the new default target.

      [root@server1 ~]# systemctl get-default
multi-user.target

      Alternatively:

      [root@server1 ~]# ls -l /etc/systemd/system/default.target
lrwxrwxrwx. 1 root root 41 Mar 10 08:00 /etc/systemd/systemd/default.target ->
/usr/lib/systemd/system/multi-user.target

In our environment, the graphical console is not available (except via VNC). If you want to set the default target back to the graphical target, use:

[root@server1 ~]# systemctl set-default graphical.target

2. Resetting a Lost Root Password

Lab Resources

Machines: server1

Goal

  • A recovered root password

In this lab, you will recover a lost root password. Assume that you do not know the root password for server1 and it must be changed.

  1. Log in to the GUI of your desktop1 system using the guacamole interface.

  2. Start Konsole and then run the virt-manager tool.

  3. In the virt-manager GUI, select the vserver VM then click Open.

    The vserver VM is your server1.example.com host.

  4. Click the Console button to see the VM’s console.

  5. Click View → Scale Display → Always to make the entire console visible in the screen.

  6. Reboot your system, and interrupt the countdown in the boot loader menu.

    1. Click Send Key → Ctrl+Alt+Del.

    2. When the boot loader menu appears, press any key to interrupt the countdown.

  7. Edit the default boot loader entry (in memory) to abort the boot process just after all file systems have been mounted, but before control is handed over to systemd, then boot.

    1. Use the cursor keys to highlight the default boot loader entry.

    2. Press e to edit the current entry.

    3. Using the cursor keys, navigate to the line that starts with linux16.

    4. Press End to move the cursor to the end of the line.

    5. Append init=/bin/bash to the end of the line.

    6. You will need to open the virtual keyboard to do the next step. To open the virtual keyboard:

      1. Mac: press Ctrl - Option - Shift

      2. Linux: press Ctrl - Alt - Shift

    7. From the virtual keyboard you can click Ctrl + x to boot using the modified configuration. Press the key combo again to close the virtual keyboard.

  8. At the bash prompt do the following:

    bash# /usr/sbin/load_policy -i
bash# mount -o remount,rw /

  9. Change the root password to something of your own choosing.

    bash# passwd root

  10. Run the exec /sbin/init command to resume the initialization and finish the system boot.

    bash# exec /sbin/init

  11. When the login prompt appears, log in to your server1 machine as root using the new root password.

  12. Reboot server1 before proceeding to the next lab.

3. Repairing Boot Problems

Lab Resources

Machines: server1 bootvm

Goal

  • Your machine boots normally, without user intervention

In this lab, you will recover from an error in /etc/fstab.

  1. Log in to the GUI of your desktop1 system using the guacamole interface.

  2. Start Konsole, then run the virt-manager tool.

  3. Create an entry in /etc/fstab that will simulate a failed boot situation, then reboot.

    [root@localhost ~]# echo "/dev/fail /fail xfs defaults 1 1" >> /etc/fstab
[root@localhost ~]# reboot

  4. Take a good look at the console of your server1 machine. It is stuck early in the boot process due to the bad fstab entry.

    1. Reboot the machine and interrupt the boot loader menu countdown.

      If you wait long enough, the system will eventually spawn a rescue shell by itself, but that can take a while, so we will spawn the rescue shell manually.

      If you use Ctrl+Alt+Del to reboot and it does not cause the system to reboot, you can either wait for the task to time out or use the VM’s power switch to force a reboot.

    2. When the boot loader menu appears after the BIOS self-test, press any key to interrupt the countdown.

  5. Use the cursor keys to highlight the default boot loader entry.

    1. Press e to edit the current entry.

    2. Using the cursor keys, navigate to the line that starts with linux16.

    3. Press End to move the cursor to the end of the line.

    4. Append systemd.unit=emergency.target to the end of the line.

    5. Press Ctrl+x to boot using the modified configuration.

  6. Log in to the emergency mode.

    Pay close attention to any errors you receive. Note that it did not hang for a long time waiting to show the password prompt. This is one of the benefits of booting in this mode.

    1. At the Give root password for maintenance prompt enter the root password.

  7. Inspect what file systems are currently mounted.

    [root@localhost ~]# mount
...
/dev/vda1 on / type xfs (ro,relatime,seclabel,attr2,inode64,noquota)

  8. It appears that the root file system is mounted read-only; mount it read-write.

    [root@localhost ~]# mount -oremount,rw /

  9. Attempt to mount all the other file systems.

    [root@localhost ~]# mount -a
mount: mount point /fail does not exist

  10. Open /etc/fstab in an editor and fix the issue.

    [root@localhost ~]# vi /etc/fstab

    1. Remove the invalid line (the one with /fail).

    2. Save your changes, then exit your editor.

  11. Verify that /etc/fstab is now correct by attempting to mount all entries.

    [root@localhost ~]# mount -a

  12. Exit your emergency shell and reboot the system by typing reboot.

    [root@localhost ~]# reboot

    Your system should now boot normally.

4. Repairing a Boot Loader Problem

Lab Resources

Machines: server1

Goal

  • A machine that boots normally without user intervention

In this lab, you will repair an issue with the boot loader configuration on one of your machines.

  1. Access the server1 system using the GUI and log in as root.

  2. Break grub with the following command then reboot the VM.

    [student@server1 ~]$ sed -i "s/linux16/os16/" /boot/grub2/grub.cfg
[student@server1 ~]$ reboot

    The system should fail to boot right after coming out of grub.

    error: can't find command `os16'.
unaligned pointer 0x3fed58bf
Aborted. Press any key to exit.

  3. Reboot the server1 VM by using Send Key → Ctrl+Alt+Del, then interrupt the boot loader countdown timer.

  4. Move the cursor to the default boot entry, then press e to edit that entry. Inspect the configuration closely, looking for text starting with os16 (inserted with the sed command earlier).

  5. Find the line that is blocking the boot process, modify it, and then boot with these changes.

    1. os16 is not a valid grub directive. Change it to linux16.

    2. Press Ctrl+x to boot your system with the modified configuration.

  6. Wait for the system to boot, log in as root, and then generate a new grub2 configuration.

    Do not immediately overwrite the existing configuration, but inspect the new configuration first.

    [root@server1 ~]# grub2-mkconfig > /tmp/newgrub

    1. Look at the differences between the broken existing grub and the new proposed configuration. (Ignore the lines about msdos.)

      [root@server1 ~]# diff /boot/grub2/grub.cfg /tmp/newgrub

    2. Commit the configuration to disk.

      [root@server1 ~]# grub2-mkconfig > /boot/grub2/grub.cfg

  7. Reboot your machine, and check if it boots normally again without user intervention.

    [root@server1 ~]# systemctl reboot